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Abstract — This paper presents a face completion and 
recognition algorithm under partial occlusion. Component- 
based algorithm utilized in the proposed method. Individual 
components are detected using Haar-like features and wild 
card estimation is used for face completion task. The system 
performs better results under partial occlusion conditions in 
comparison to existing methods. AR face database is used 
for experiments. 

Index Terms — Face recognition, face completion, occlusion, 
Haar-features, topology, components. 



I. 



INTRODUCTION 



Face recognition techniques are user friendly and useful to 
identify people without disturbance. Under constrained 
conditions most of the face recognition systems give better 
result results as compared to natural conditions. When images 
are associated with occlusions, varying illumination, pose 
variation, the result of face recognition systems decrease. 
Presently partial occlusion is a huge problem for face 
recognition, which comes due to scarf, sunglasses. This paper 
explains how to complete face, which is associated with 
occlusion or some components are missing. 
Main steps for the completion of face are: individual 
components of occluded face are detected by using Viola and 
Jones extension proposed by Lienhart et al. [1] [2], and faces 
from database are used to match with these components using 
AdaBoost ensemble learning. Euclidean distance can be used 
for this purpose also. Finally occluded face completed by 
using graph matching techniques. 

There are many other techniques for completion of face, e.g. 
graph-cuts, Poisson equations and inpainting [3] [4], but these 
techniques are useful only for high-resolution color images 
and more information needed in case of damaged images. 
Available face recognition algorithms are useful for gray 
images and these algorithms will decrease in their 
performance. Our proposed method overcomes all these 
problems and gives better results. 
This paper has two major contributions: 

1) Our proposed method used component detection and graph 
matching for completion of face and different from existing 
methods. 

2) Reconstruction rate increases by using this proposed 
method. 



II. RELATED WORK 

Detection of objects from images and videos are the main 
purpose of many available techniques. There are two types of 
systems component-based and holistic approaches. Individual 
components are used for detection in component-based 
methods and holistic approaches detect an object as a whole. 
Diverse and original face completion techniques are reviewed 
in this section. Only seminal work of component-based and 
holistic methods is reviewed in this section and ignores 
extensions. Features extracted from different components, 
type of target, and recognition approach are the comparison 
points for these two methods. Topology verification can be 
performed by different techniques, in case of component- 
based approach. Table I gives comparison of recent object 
detection systems in terms of several characteristics. 

TABLE I 

Comparison of Object Detection Systems 



Name 


Features 


Classifier 


Topology 


Heisele [5] 


Pixel values 


SVM 


SVM 


Mohan [6], [7] 


Haar-wavelets 


SVM 


SVM 


Rowley [8], [9] 


Pixel values 


Neural 
networks 




Schneiderman 
[10], [11] 


PCA 

coefficients, 
wavelets 


Bayes 




Felzenszwalb 
[12] 


Derivations of 
Gaussians 


Energy 
function 


Energy 
function 


proposed 


Haar-like 


AdaBoost 


Matching of 
graph 



Eigenfaces are used by Turk and Pendland to completion of 
face [13]. To remove eyeglasses PCA -based algorithm is used 
by Saito et al. [14]. Due to the occluded pixels reconstructed 
images are distorted. Damaged photos reconstruction uses 
stochastic gradient-based technique proposed by Jone and 
Poggio [15]. Due to many steps the process is slow. 
Feature points are manually selected in technique proposed 
by Hwang et al. [16]. The other method proposed by them is 
based on a morphable face model [17]. They use least-square 
minimization (LSM) to get optimal coefficients from 
undamaged parts of face and this method is based on PCA 
also. An efficient error correction algorithm proposed by 
Wright et al. is robust PCA (RPCA) [18]. In this technique, 
low-rank structure of face presented using nuclear norm. 
Internet images are used for face completion in a technique 
presented in recent study [19]. It gives good performance. 
Graphic area [20] uses Poisson equation [21] for completion 
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IJSEcls. To perform experiments gray and small images are 
used by face completion techniques. Small images are not 
affected by Poisson equation. Gabor features used by Zhang 
et al. for face recognition under occlusion conditions. Gabor- 
based feature [22] gives good experimental results, between 
test image and reference images. This method used 
perceptual-distance-based function and kernel LDA classifier. 
Small training set is used by Lin et al. occluded face 
recognition. In this method the effect of occlusion reduces by 
focusing on matched local regions, and Gaussian mixture 
model is used for similarity feature [23]. 
Under occlusion conditions SR has done good work, e.g., 
[24], [25], and [26]. According to this paper certain 
probability distribution has followed by occluded part. For 
occlusion condition Gabor-filter based technique represented 
by Yang and Zhang. Computational cost can be reduces by 
the combination of gabor wavelet and SR [27]. 

III. SYSTEM DESCRIPTION 

Component based approach is used for the development of 
proposed system. The first step of proposed method is 
detection of individual components, than face localization 
step is performed. Proposed system overview is presented in 
fig. 2. Statistical and structural techniques are used for the 
development of proposed system. Statistical concept is used 
for component detection and structural concept is used for 
topology verification. 



Individual 




Graph 




Wildcard 




Face 


component 




construe 




estimation 




localizat- 


detection 




tion 








ion 



Fig. 2. Overview of the system. 




(a) (b) (c) 




(d) (e) 
Fig. 3. Example of different parts of the system, (a) Input 
face, (b) Component detection, (c) Graph matching, (d) 
Wildcard estimation, (e) Localization of face. 

Individual parts of the system are shown in fig. 3. Fig. 3(a) 
shows occluded input image. Fig. 3(b) shows individual 
components which are detected by component detection. 
Mouth and nose are occluded, so can't detect. Graph 
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matching results shown in fig. 3(c). Black box is used to 
highlight the detected components and white box is used to 
insert wildcard components. Fig. 3(d) shows the wildcard 
components. In Fig. 3(e) individual components are used to 
face localization. Component detector can detect only two 
components and the proposed approach is used to completion 
of the face. 

A. DETECTION OF INDIVIDUAL PARTS 

Individual components such as mouth (M), nose (N), and 
right eye (R), left eye (E) are localized by the component 
detection step. Different components are detected by 
individual detectors. Training data is different for each 
detector, but machine learning approach is same. 
Many classification methods presented in past are used by the 
face detection systems. Many authors use Viola and Jones [1] 
method for detection purpose, because it is fast and robust. 
We use extension of Viola and Jones proposed by Lienhart et 
al. [2]. 

The detection of objects is performed by different block size 
which moves row-wise. The classifier takes decision about 
the object based on the features extracted by the each block. 
Actual block is normalized for illumination changes in 
processing. Block contents explain by extraction of Haar-like 
features. Extended [2] and standard [1] set of Haar-like 
features are used. These features extracted from integral 
image. AdaBoost ensemble learning strategy is used to 
trained classifier cascade. Many strong classifiers are used to 
design cascade. Many weak classifiers are used to develop 
strong classifier. These weak classifiers are combined using a 
weighted summation. One Haar-like feature with parity and 
threshold is consisted by each weak classifier. AdaBoost is 
used to choose suitable weak classifiers. Object of interest 
represented by actual block is decided by strong classifier at 
each cascade stage, discarded the irrelevant blocks and passes 
the relevant blocks to next stage. Block contains the object if 
it passes the last stage. Classifier cascade improves speed as 
compared to monolithic classifier. Filter heuristics is used for 
multiple overlapping detections in case of individual 
component detection. Multiple detections handle by graph- 
based topology naturally, so this step is not used in proposed 
system. 

B. GRAPH-BASED TOPOLOGY 

Proposed system explains how object detection is performed 
by using topology of individual components. Statistical 
approach is used to relate individual components by the 
component based systems. For example Mohan [7], and 
Heisele [5], [28] use support vector machine (SVM) 
approach. Occluded components can't detect by most 
statistical approaches and these approach required fixed 
features. By using topological information graph matching 
and structural pattern recognition can handle these problems. 
Fischler and Elschlager [29] approaches can be compared 
with our proposed approach. 
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US<fpoled method uses graph G (V, E), V stands for a finite, 
non empty set of vertices, and E stands for and a finite set of 
edges. Facial components define by nodes v e V, er v is size, 
and K v is component type. Euclidian distance S e = Sy i is 
defined by edge e e E, between components i and j centers. In 
matching step ratios are used instead of distance and size. 
Exact graph matching and inexact (error-tolerant) graph are 
two types of classical graph matching. Edges and nodes levels 
should be same in case of exact graph matching and level 
difference allowed for inexact graph case. 

1) Face Graph: Graph G R = (V*. E R ) is used to 
represent the face. 400 example graphs and geometric mean 
of node sizes and edge distances are used to design face 
graph. Size and position of components are used to design 
each example graph. Fig. 4 shows the resulting face graph. 




Fig. 4. Face graph (node size and edge distances compute 
manually). 

2) Designing of Graph: Firstly graph G D is constructed 
from detected components. A node v s V° is face 
components and an edge e s E D is used to connect nodes. 
Deviation is represented by the two measures of each node 
pairs. The representation of size deviation is as 



iouH 



f 17 1 
j.i' 



Li/ 3% > 1 



(1] 



n75 



graph G extracted in many graphs Gf = (Vf , Ef) and each 
graph represent single face. 

4) Matching of Graph: There is different size and 
location of components consisted by each graph G c Matching 
process is done and matched subgraph find out in this step. 
Low similarity combinations should be removed. 

A component v e V w , without size and distance information 
is introduced for completion of face. All subgraphs Gf = 

(Vi,Ei) with minimum two different detected components 

are selected from the resulting graph. 

5) Wildcard Estimation: Several wildcard components 
are included in the resulted matched graph after the graph 
matching step. Facial components coordinates are required to 
estimate the face region. Reference graph, orientation 
information and detected components are used to estimate 
missing coordinates. The determinant of P[ = (X] y{) T , = ( x 2 
y2) T , P3 = (x3 y3) T , is used to compute the orientation 

, i /xl — x3 x2 — x3\ I 
d6t= I U->"3 J'2-ysJ I 



Coordinate P 3 is estimated for the wildcard component v?. 
6) Face Localization: Subgraph components and 
reference graph are used to estimate the face region. 
Subgraph components are left eye L, right eye R, nose N, 
and mouth M. The rectangle b= (x, y, w, h) is used to 
describe the face region, where w is width, h is height and 
(x, y) T is center. 

w = 2.(x R -x L ) 

X = X N 

y = yn 



Representation of distance deviation is as 



*0 



-1, if 



■> 1 



■irr[Lj 

1 J if Srfcjrlfi ^ 



3) Labeling of Connected Components: Many 
connected components of face are consisted by graph G D The 




it 



Fig. 5. Face completion examples. 
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IV. EXPERIMENTS 



A lot of images are required for the development of face 
recognition system. These images are used to generate model 
during training and to obtain general results during testing. 
There are many databases available but selection of suitable 
images for specific face recognition system is difficult. 
BioID Face Database [30] and AR Face Database [31] are 
used for training. 1521 images of BioID Face Database and 
508 images of AR Face Database are trained for positive 
samples. For negative samples, 19973 images are taken from 
Data Becker 222222 image gallery. 143 images from AR 
Face Database and Corel Gallery Magic 65000 are used for 
testing. 

The first stage of proposed face recognition system is facial 
component detection. Extended set of haar features are used 
for the development of component detector. 4000 images of 
mouth and nose are used for training of positive samples, and 
2000 images of each eye are used. 3000 images are used for 
training of negative samples. Individual component 
uniqueness is the reason of difference, because number of 
training images is same. 

V. CONCLUSION 

In this paper, we have proposed a wild card estimation and 
topology verification method for completion of occluded face 
and partial distance measure for recognition task. Viola and 
Jones approach is used to detect individual components. 
Specific graph model is used to describe the topology of 
different components. Size and distance ratios are considered 
by graph model. The recognition rate improved under partial 
occlusion of scarf and glass by proposed method. Proposed 
face recognition algorithm has limitation of resolution, so 
future work will concentrate to overcome the resolution 
limitation automatic estimation of components. 
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